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Introduction 

What Is PIAAC? 

The Program for the International Assessment of Adult Competencies (PIAAC) is a cyclical, 
large-scale study of adult skills and life experience focusing on education and employment that was 
developed and organized by the Organization for Economic Cooperation and Development (OECD). 
In the United States, the study was conducted in 201 1-12 with a nationally representative sample of 
5,000 adults between the ages of 16 and 65. Similar nationally representative samples of adults were 
surveyed in each of the 22 other participating countries. 1 

The goal of PIAAC is to assess and compare the basic skills and the broad range of competencies of 
adults around the world. The assessment focuses on cognitive and workplace skills necessary for 
successful participation in 21st-century society and the global economy. Specifically, PIAAC measures 
relationships between individuals’ educational background, workplace experiences and skills, occupa- 
tional attainment, use of information and communication technology, and cognitive skills in the areas 
of literacy, numeracy, and problem solving in technology-rich environments. 

History of International Adult Literacy Assessments 

PIAAC builds on knowledge and experiences gained from previous international adult assessments — 
the International Adult Literacy Survey (IALS) and the Adult Literacy and Lifeskills Survey (ALL). 
Statistics Canada initiated both of these earlier assessments. PIAAC improves upon the content frame- 
works of these assessments as well as their design and methodologies. PIAAC seeks to ensure continu- 
ity with previous literacy surveys and includes items from IALS and ALL. PIAAC improves on quality 
assurance standards, extends the definitions of literacy and numeracy, presents the problem-solving 
domain to emphasize skills used in technology-rich environments, and provides more information 
about individuals with low levels of literacy by assessing reading component skills. 

What Does PIAAC Measure? 

PIAAC is designed to assess adults in different countries over a broad range of abilities, from simple 
reading to complex problem-solving skills. To do this, PIAAC defines four core competency domains 
of adult cognitive skills that are seen as key to facilitating the social and economic participation of 
adults in advanced economies: literacy, reading components, numeracy, and problem solving in 
technology-rich environments. All participating countries and regions are required to assess the literacy 
and numeracy domains, but the reading components and problem solving in technology-rich environ- 
ments domains are both optional. The United States assessed all four domains. Lor a list of the subject 
experts for each of those domains, see appendix A. Lor a more detailed description of the four domains, 
see appendix B. 

PIAAC tasks developed for these domains are authentic, culturally appropriate, and drawn from 
real-life situations that are expected to be important or relevant in different contexts. Tasks are intended 
to be reflective of adults’ daily lives across cultures, even if not every adult is necessarily familiar with 
every task. PIAAC is not designed to provide individual scores, but rather it measures how groups of 
adults perform on the domains. Each respondent takes only a portion of the items in each domain. 


1 Data for the Russian Federation are not included in this First Look report because they were received too late for publication. 


Literacy 

The PIAAC literacy framework expands on the definition of literacy previously used in IALS and ALL. 
PIAAC broadly defines literacy as “ understanding, evaluating, using and engaging with written text to 
participate in society ; to achieve ones goals and to develop ones knowledge and potential” (Organization for 
Economic Cooperation and Development [OECD], 2012.) 

Reading components 

The primary goal of the PIAAC reading components measure is to provide information about the 
literacy skills of adults at the lower end of the literacy spectrum — specifically, whether they have the 
foundational skills to develop the higher literacy and numeracy abilities necessary for functioning in 
society. The reading components assessment focuses on elements of reading that are comparable across 
the range of languages in the participating countries: reading vocabulary, sentence comprehension, 
and basic passage comprehension. 

Numeracy 

The primary goal of PIAAC s numeracy assessment is to evaluate basic mathematical and computational 
skills that are considered fundamental for functioning in everyday work and social life. Numeracy in 
the PIAAC framework is defined as u the ability to access , use, interpret, and communicate mathematical 
information and ideas, to engage in and manage mathematical demands of a range of situations in adult 
life” (Organization for Economic Cooperation and Development [OECD], 2012.) 

Problem solving in technology-rich environments 

PIAAC represents the first attempt to assess problem solving in technology-rich environments on a 
large scale and as a single dimension in an international context. PIAAC defines problem solving in 
technology-rich environments as “ using digital technology, communication tools, and networks to acquire and 
evaluate information, communicate with others, and perform practical tasks” (Organization for Economic 
Cooperation and Development [OECD], 2012.) 

Skill use and the background questionnaire 

In addition to focusing on the direct measurement of adult competencies in the three main cognitive 
domains of literacy, numeracy, and problem solving in technology-rich environments, PIAAC also 
examines adults’ intrapersonal, interpersonal, and professional skills through a background questionnaire. 
For a more detailed description of the skill use and the background questionnaire, see appendix C. 
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Reporting Results 

PIAAC results are reported in two ways: as scale scores on a 0-500 scale in three domains (literacy numeracy 
and problem solving in technology-rich environments), 2 and as percentages of adults reaching established 
proficiency levels. PIAAC reports five proficiency levels for literacy and numeracy (Below level 1, Level 1, 

Level 2, Level 3, and Level 4/5) and four levels for problem solving in technology-rich environments (Below 
level 1, Level 1, Level 2, and Level 3). Across all countries, only 2 percent of adults performed at Level 5 on 
many of the variables in the literacy and numeracy scales. This report follows OECD reporting conventions 
by combining the top two proficiency levels for the literacy and numeracy scales. There were fewer items 
in the problem solving in technology-rich environments assessment to define the scale, which accounts for 
why there are fewer proficiency levels in this scale. Appendix B provides information about interpreting the 
proficiency levels. 

The purpose of this First Look report is to introduce PIAAC data through the presentation of data 
figures and tables and selected findings. 3 However, readers are cautioned not to draw causal inferences. 
Many of the variables examined in this report may be related to one another, but the complex inter- 
actions and relationships among them have not been explored. The variables examined here are also just 
a few of the variables that can be examined in these data; they were selected to demonstrate the range of 
information available from the study. The release of this report is intended to encourage more in-depth 
analysis of the data using more sophisticated statistical methods. 

All statistically significant differences described in this report are at the .05 level. No statistical adjustments 
to account for multiple comparisons were used. Differences that are statistically significant are discussed 
using comparative terms such as “higher” and “lower.” Differences that are not statistically significant are 
either not discussed or referred to as “not measurably different” or “not statistically significant.” In the 
latter case, failure to find a difference as statistically significant does not necessarily mean that there was 
no difference. It could be that a real difference cannot be detected by the significance test because of small 
sample size or imprecise measurement in the sample. If the statistical test is significant, this means that 
there is convincing evidence (though no guarantee) of a real difference in the population. However, it is 
important to remember that statistically significant results do not necessarily identify those findings that 
have policy significance or practical importance. See appendix C for more information about statistical 
testing. 

Trend Results 

This report presents overall trend comparisons over time for the total adult population in the areas 
of literacy and numeracy. In literacy, comparisons are made between PIAAC (2012) and both ALL 
(2003-2008) and IALS (1994-1998). In numeracy, trend comparisons are made between PIAAC 
(2012) and ALL (2003-2008). In both the literacy and numeracy domains, approximately 60 percent 
of the items are common between PIAAC and previous international surveys to ensure the compara- 
bility of these domains. 


2 Results from the reading components portion of the assessment are not reported separately in this report, but can be accessed from the International Data Explorer 
(IDE) at http://www.oecd.org/site/piaac/publicdataandanalysis.htm. These results have not yet been analyzed internationally, but will be part of a forthcoming report 

from the OECD, which will be followed by an analysis of how U.S. adults performed on this domain. 

3 Selected findings are presented for only some figures. 
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Selected Findings 

• Average scores on the PIAAC literacy scale for adults age 16 to 65 ranged from 250 in Italy to 
296 in Japan. The U.S. average score was 270. Compared with the U.S. average score, average 
scores in 12 countries were higher, in 5 countries they were lower, and in 5 countries they were 
not significantly different (see figure 1-A). 

• Average scores on the PIAAC numeracy scale for adults age 16 to 65 ranged from 246 in Spain 
to 288 in Japan. The U.S. average score was 253. Compared with the U.S. average score, aver- 
age scores in 1 8 countries were higher, in 2 countries they were lower, and in 2 countries they 
were not significantly different (see figure 1-B). 

• Average scores on the PIAAC problem solving in technology-rich environments scale for adults 
age 16 to 65 ranged from 275 in Poland to 294 in Japan. The U.S. average score was 2 77. 
Compared with the U.S. average score, average scores in 14 countries were higher and in 

4 countries they were not significantly different (see figure 1-C). 

• Twelve percent of U.S. adults age 16 to 65 performed at the highest proficiency level (4/5) on 
the PIAAC literacy scale. The percentage of adults performing at this level was higher than in 
the U.S. in 7 countries (Japan, Finland, Netherlands, Australia, Sweden, Norway, and Canada), 
lower in 1 1 countries (Denmark, Poland, Czech Republic, Austria, France, Ireland, Republic of 
Korea, Slovak Republic, Cyprus, Spain, and Italy), and not significantly different in 4 countries 
(England and Northern Ireland-United Kingdom, Flanders-Belgium, Estonia, and Germany) 
(see figure 2-A). 

• Nine percent of U.S. adults age 16 to 65 performed at the highest proficiency level (4/5) on 
the PIAAC numeracy scale. The percentage of adults performing at this level was higher than 
in the U.S. in 15 countries (Finland, Japan, Sweden, Flanders-Belgium, Norway, Netherlands, 
Denmark, Germany, Australia, Austria, Slovak Republic, Canada, Czech Republic, England and 
Northern Ireland-United Kingdom, and Estonia), lower in 3 countries (Republic of Korea, 

Italy, and Spain), and not significantly different in 4 countries (France, Poland, Ireland, and 
Cyprus) (see figure 2-B). 

• Six percent of U.S. adults age 16 to 65 performed at the highest proficiency level (3) on the 
PIAAC problem solving in technology-rich environments scale. The percentage of adults 
performing at this level was higher than in the U.S. in 8 countries (Japan, Finland, Sweden, 
Czech Republic, Canada, Germany, Netherlands, and Australia), lower in 2 countries (Ireland 
and Slovak Republic), and not significantly different in 8 countries (Poland, Denmark, 
Flanders-Belgium, Norway, England and Northern Ireland-United Kingdom, Estonia, Austria, 
and Republic of Korea) (see figure 2-C). 

• The percentage of U.S. adults between 55 and 65 years old who scored at the highest 
proficiency level (4/ 5) on the PIAAC literacy scale was higher than the international average 
for this age group. For all other age groups, the percentage of adults performing at the highest 
proficiency level on the PIAAC literacy scale was not significantly different for U.S. adults and 
the international average for their peers (see figure 3-A). 
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The percentage of U.S. adults between 55 and 65 years old who scored at the highest 
proficiency level (4/5) on the PIAAC numeracy scale was not significantly different than the 
international average for this age group. For all other age groups, the percentage of adults 
performing at the highest proficiency level (4/5) on the PIAAC numeracy scale was lower 
for U.S. adults than the international average for their peers (see figure 3-B). 

The percentages of U.S. adult males and U.S. adult females age 16 to 65 performing at the 
highest proficiency level (4/5) on the PIAAC numeracy scale were lower than the international 
averages for their peers at this level (see figure 4-B). 

The percentage of adults age 16 to 65 born in the United States who performed at the highest 
proficiency level (4/5) on the PIAAC literacy scale was higher than the percentage of adults 
born outside the United States who performed at this level (see figure 5-A). 

The percentage of U.S. adults age 16 to 65 with graduate or professional degrees who 
performed at the highest proficiency level (4/5) on the numeracy scale was lower than the 
international average for adults with graduate or professional degrees who performed at this 
level (see figure 6-B). 

The percentage of White U.S. adults age 16 to 65 at the highest proficiency level (4/5) on the 
literacy scale was higher than the percentages of their Black or Hispanic peers at this level 
(see figure 7- A). 

Among adults age 16 to 65 who use information and communication technology (ICT) 
skills the most at work (top quintile), the percentage of U.S. adults performing at the 
highest proficiency level (3) on problem solving in technology-rich environments was not 
significantly different than the international average for adults who use ICT skills most at 
work (see figure 8-C). 

The percentage of employed U.S. adults age 16 to 65 performing at the highest proficiency 
level (4/5) on the PIAAC numeracy scale was lower than the international average of employed 
adults at this level (see figure 9-B). 

The percentage of U.S. adults age 16 to 65 with excellent health performing at the highest 
proficiency level (3) on the problem solving in technology-rich environments scale was higher 
than their peers who reported having poor, fair, or good health at this level (see figure 10-C). 

In the United States, there was no statistical difference in the percentage performing at the 
highest proficiency level (3) on problem solving in technology-rich environments between 
adults age 16 to 65 who reported having a learning disability and those who reported not 
having a learning disability (see figure 1 1-C). 

On the literacy scale, the average score for adults age 16 to 65 in the United States on PIAAC 
(2012) was not significantly different from the score for U.S. adults on ALL (2003-08), but 
was lower than the score for U.S. adults on IALS (1994-98) (see table 1-A). 

On the numeracy scale, the average score for adults age 16 to 65 in the United States was lower 
on PIAAC (2012) than on ALL (2003-08) (see table 1-B). 
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Figure 1 -A. Average scores on the PIAAC literacy scale for adults age 1 6 to 65, by participating country and region: 201 2 
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Adult Competencies (PIAAC), 2012. 
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Figure 1 -B. Average scores on the PIAAC numeracy scale for adults age 1 6 to 65, by participating country and region: 201 2 
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Figure 1 -C. Average scores on the PIAAC problem solving in technology-rich environments scale for adults age 16 to 65, 
by participating country and region: 2012 


Japan 

Finland 

Australia 

Sweden 

Norway 

Netherlands 

Austria 

PIAAC international average 

Denmark 
Czech Republic 
Republic of Korea 
Germany 
Canada 
Slovak Republic 

Flanders-Belgium 

England and Northern Ireland- 
United Kingdom 

Estonia 

United States 

Ireland 

Poland 



Scale score 

J Score is significantly □ Score is not significantly Q Score is significantly 

higher than the different from the lower than the 

U.S. average score U.S. average score U.S. average score 

NOTE: Countries and regions are listed in descending order determined by their unrounded average scores. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of 
Adult Competencies (PIAAC), 2012. 


9 


Figure 2-A. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, by participating 
country and region: 2012 
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Figure 2-B. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, by participating 
country and region: 2012 
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than is shown in the whole numbers displayed. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of 
Adult Competencies (PIAAC), 2012. 
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Figure 2-C. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC problem solving in technology-rich 
environments scale, by participating country and region: 2012 
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than is shown in the whole numbers displayed. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of 
Adult Competencies (PIAAC), 2012. 
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Figure 3-A. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, by 10-year age intervals: 2012 
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Figure 3-B. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, by 10-year age intervals: 2012 
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Figure 3-C. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC problem solving in technology-rich 
environments scale, by 10-year age intervals: 2012 
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1 Country- and region-specific results are available at http://www.oecd.org/site/piaac/publicdataandanalysis.htm. 

NOTE: Percentages of adults age 16 to 65 by 10-year age intervals appear in parentheses. Due to differences in the sampling and data collection, the percentage distribution for the categories in the problem solving in 
technology-rich environments scale may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more detailed information). Detail may not sum to totals because of 
rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of 
Adult Competencies (PIAAC), 2012. 
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Fig u re 4-A. Percentage of adults age 1 6 to 65 at each level of proficiency on the PI AAC literacy scale, by gender: 201 2 
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Figure 4-B. Percentage of adults age 16 to 65 at each level of proficiency on the PI AAC numeracy scale, by gender: 2012 
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Figure 4-C. Percentage of adults age 16 to 65 at each level of proficiency on the PI AAC problem solving in technology-rich 
environments scale, by gender: 2012 
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1 Country- and region-specific results are available at http://www.oecd.org/site/piaac/publicdataandanalysis.htm. 

NOTE: Percentages of adults age 16 to 65 by gender appear in parentheses. Due to differences in the sampling and data collection, the percentage distribution for the categories (e.g., male and female) in the problem 
solving in technology-rich environments scale may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more detailed information). Detail may not sum to totals 
because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of 
Adult Competencies (PIAAC), 2012. 
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Fig u re 5-A. Percentage of U.S. adults age 1 6 to 65 at each level of proficiency on the PIAAC literacy scale, by 
whether they were born in the United States: 201 2 
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Figure 5-B. Percentage of U.S. adults age 1 6 to 65 at each level of proficiency on the PIAAC numeracy scale, by 
whether they were born in the United States: 201 2 


20 


35 


No (15) 

Overall 


1 

23 

' L . 

25 

1 

27 

18 

6 

10 

J 

20 

J 

34 

| 

27 

9 


0 10 20 30 40 50 60 70 80 

Percent 

| | Below level 1 Q Level 1 Q Level 2 Q Level 3 | Level 4/5 


90 


100 


Figu re 5-C. Percentage of U.S. adults age 1 6 to 65 at each level of proficiency on the PIAAC problem solving in 
technology-rich environments scale, by whether they were born in the United States: 201 2 
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NOTE: Percentages of adults age 16 to 65 by whether they were born in the United States appear in parentheses. Due to differences in the sampling and data collection, the percentage distribution for the categories in 
the problem solving in technology-rich environments scale may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more detailed information). Detail may not 
sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), 2012. 
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Figure 6-A. Percentage of adults age 1 6 to 65 at each level of proficiency on the PIAAC literacy scale, by highest level of 
educational attainment: 2012 
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Figure 6-B. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, by highest level of 
educational attainment: 2012 


United States 

professional dlcfree (10) 
Bachelor's degree (16) 
Associate's degree (9) 
High school credential (50) 
Below high school (15) 
Overall 

PIAAC international average 1 

Graduate or 

professional degree (10) 
Bachelor's degree (11) 
Associate's degree (11) 
High school credential (46) 
Below high school (24) 
Overall 



10 


20 


34 



10 


20 


30 


40 


50 

Percent 


60 


70 


80 


90 


. — . Below 
I I level 1 

□ Level 1 

□ Level 2 

□ Level 3 

□ Level 4/5 


100 


Fig u re 6-C. Percentage of adults age 1 6 to 65 at each level of proficiency on the PIAAC problem solving in technology-rich 
environments scale, by highest level of educational attainment: 2012 
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1 Country- and region-specific results are available at http://www.oecd.org/site/piaac/publicdataandanalysis.htm. 

NOTE: Percentages of adults age 16 to 65 by highest level of educational attainment appear in parentheses. Due to differences in the sampling and data collection, the percentage distribution for the categories in the 
problem solving in technology-rich environments scale may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more detailed information). Detail may not sum to 
totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of 
Adult Competencies (PIAAC), 2012. 
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Fig u re 7-A. Percentage of U.S. adults age 1 6 to 65 at each level of proficiency on the PIAAC literacy scale, by 
race/ethnicity: 2012 
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Figure 7-B. Percentage of U.S. adults age 1 6 to 65 at each level of proficiency on the PIAAC numeracy scale, by 
race/ethnicity: 2012 
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Figure 7-C. Percentage of U.S. adults age 16 to 65 at each level of proficiency on the PIAAC problem solving in 
technology-rich environments scale, by race/ethnicity: 2012 


White (69) 

C 

14 | 


41 

1 

37 


Black (12) 

C 


38 

1 

45 

1 

15 

Hispanic (11) 

c 


37 

l 

41 

1 

21 | | 

Other (8) 

c 

21 

1 

42 

1 

32 

. LJ 

Overall 

[Z 

20 

1 

41 

1 

33 

MM 


0 

10 

20 30 

40 

50 60 

70 80 

90 100 






Percent 




I Below level 1 

I | Level 1 

I | Level 2 

I | Level 3 




NOTE: Percentages of adults age 16 to 65 by race/ethnicity appear in parentheses. Due to differences in the sampling and data collection, the percentage distribution for the categories in the problem solving in 
technology-rich environments scale may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more detailed information). Black includes African American, and 
Hispanic includes Latino. Race categories exclude Hispanic origin. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), 2012. 
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Figure 8-A. Percentage of adults age 1 6 to 65 at each level of proficiency on the PIAAC literacy scale, by frequency with which reading 
skills are used at work: 201 2 
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Figure 8-B. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, by frequency with which 
numeracy skills are used at work: 201 2 


United States 


Top quintile (21) | 3 | 


14 


31 



Bottom quintile (13) 

Overall 

PIAAC international average 1 


22 


37 


10 


20 


34 


Top quintile (15) H 5 | 22 

Upper middle quintile (15) |1| 8 | 

Middle quintile (16) |1| 9 | 

Lower middle quintile (16) | 3 | 13 f - 

Bottom quintile (16) | 4 | 14 

Overall | 5 | 14 



, — , Below 
I I level 1 

□ Level 1 

□ Level 2 
I Level 3 
I Level 4/5 


0 10 20 30 40 50 60 70 80 90 100 

Percent 


Figure 8-C. Percentage of adults age 1 6 to 65 at each level of proficiency on the PIAAC problem solving in technology-rich environments 
scale, by frequency with which information and communication technology (ICT) skills are used at work: 201 2 
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1 Country- and region-specific results are available at http://www.oecd.org/site/piaac/publicdataandanalysis.htm. 

NOTE: A scale of "skill use at work" was created from information that adults age 16 to 65 reported about the frequency with which they used particular skills at work. For each of the skills asked about, respondents who are currently 
employed could report using it "never," "less than once a month," "less than once a week but at least once a month," "at least once a week but not every day," or "every day" at work. Based on these responses, a scale was created using a 
generalized partial credit model (GPCM) to quantify the distribution of skill use at work. The top quintile of this scale indicates adults whose skill use at work is at the highest reported levels; the bottom quintile of this scale indicates 
adults whose skill use at work is at the lowest reported levels. Percentages of adults age 16 to 65 who fall into each of the quintiles are reported in parentheses to the right of the quintile labels. The quintile boundaries are based on the 
international PIAAC database, so they do not contain exactly 20 percent in each of the countries. Respondents who reported that they never used a skill on any of the items that formed part of the set of reading skills used at work, 
numeracy skills used at work, or ICT skills used at work constitute a separate category not shown here. For a list of the skills in each of these three sets of skills used at work, see appendix C. Detail may not sum to totals 
because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of Adult Competencies (PIAAC), 2012. 
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Figure 9-A. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, by employment status: 2012 
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Figure 9-B. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, by employment status: 2012 
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Figu re 9-C. Percentage of adults age 1 6 to 65 at each level of proficiency on the PIAAC problem solving in technology-rich 
environments scale, by employment status: 201 2 
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1 Country- and region-specific results are available at http://www.oecd.org/site/piaac/publicdataandanalysis.htm. 

NOTE: Percentages of adults age 16 to 65 by employment status appear in parentheses. Due to differences in the sampling and data collection, the percentage distribution for the categories in the problem solving in 
technology-rich environments scale may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more detailed information). The unemployed comprise all persons 
above a specified age who during the reference period were in the following categories: without work, that is, were not in paid employment or self-employment during the reference period; currently available for work, 
that is, were available for paid employment or self-employment during the reference period; and seeking work, that is, had taken specific steps in a specified recent period to seek paid employment or self-employment. 
Out of the labor forces comprise all persons not classified as either employed or unemployed. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of 
Adult Competencies (PIAAC), 2012. 
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Figure 1 0-A. Percentage of U.S. adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, 
by self-reported health status: 201 2 
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Figure 1 0-B. Percentage of U.S. adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, 
by self-reported health status: 201 2 
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Figure 1 0-C. Percentage of U.S. adults age 16 to 65 at each level of proficiency on the PIAAC problem solving 
in technology-rich environments scale, by self-reported health status: 2012 
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NOTE: Percentages of adults age 16 to 65 by self-reported health status appear in parentheses. Due to differences in the sampling and data collection, the percentage distribution for the 
categories in the problem solving in technology-rich environments scale may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more 
detailed information). Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), 2012. 
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Figu re 1 1 -A. Percentage of U.S. adults age 1 6 to 65 at each level of proficiency on the PIAAC literacy scale, 
by their responses to a question about whether they have ever been diagnosed or identified 
as having a learning disability: 2012 
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Figure 1 1 -B. Percentage of U.S. adults age 16 to 65 at each level of proficiency on the PIAAC numeracy 
scale, by their responses to a question about whether they have ever been diagnosed or 
identified as having a learning disability: 2012 
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Figure 1 1 -C. Percentage of U.S. adults age 16 to 65 at each level of proficiency on the PIAAC problem 
solving in technology-rich environments scale, by their responses to a question about 
whether they have ever been diagnosed or identified as having a learning disability: 201 2 
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NOTE: Percentages of adults age 16 to 65 by their response to a question about whether they have been diagnosed or identified as having a learning disability appear in 
parentheses. Due to differences in the sampling and data collection, the percentage distribution for the categories in the problem solving in technology-rich environments scale 
may differ from the percentages for the same categories in the literacy and numeracy scales (see appendix C for more detailed information). Detail may not sum to totals 
because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), 2012. 
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Table 1 -A. Average scores on the IALS, ALL, and PIAAC literacy scales for adults age 16 to 65, by participating country: Various 


years, 1994-2012 


IALS (1994- 

-1998) 

ALL (2003- 

-2008) 

PIAAC (2012) 

Country 

Scale score 

Country 

Scale score 

Country 

Scale score 

Norway 

294* 

Norway 

295* 

Netherlands 

284 

Netherlands 

286 

Netherlands 

281* 

Australia 

280 

Canada 

279* 

Canada 

280* 

Norway 

278 

United States 

273* 

Australia 

277* 

Canada 

273 

Australia 

272* 

United States 

268 

United States 

270 

Italy 

243* 

Italy 

226* 

Italy 

250 


* Significantly different (p < .05) from PIAAC (2012). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Statistics Canada 
and Organization for Economic Cooperation and Development (OECD), International Adult Literacy 
Survey (IALS), 1994-98, Adult Literacy and Lifeskills Survey (ALL), 2003-08, Program for the 
International Assessment of Adult Competencies (PIAAC), 2012. 



Score is significantly higher than the U.S. average score 
Score is not significantly different from the U.S. average score 
Score is significantly lower than the U.S. average score 


Table 1 -B. Average scores on the ALL and PIAAC numeracy scales for adults age 1 6 to 65, by participating 


country: Various years, 2003-12 


ALL (2003-2008) 

PIAAC (2012) 

Country Scale score 

Country Scale score 

Netherlands 287* 

Norway 286* 

Canada 272* 

Australia 272* 

Netherlands 280 

Norway 278 

Australia 268 

Canada 265 

United States 262* 

United States 253 

Italy 231* 

Italy 247 

* Significantly different (p < .05) from PIAAC (2012). 

SOURCE: U.S. Department of Education, National Center for 
Education Statistics, Statistics Canada and Organization for 
Economic Cooperation and Development (OECD), Adult Literacy 

Score is significantly higher than the U.S. average score 
Score is significantly lower than the U.S. average score 


and Lifeskills Survey (ALL), 2003-08, Program for the 
International Assessment of Adult Competencies (PIAAC), 2012. 
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More Information About PIAAC 

This report is a first look at the PIAAC 2012 results and provides findings for only a few select results. 

For more PIAAC 2012 results and information about the assessment: 

•Preview and print a selection of data on the performance of U.S. adults on the PIAAC assessment 
for various topics across all three domains: http://nces.ed.gov/surveys/piaac/index.asp . 

•Find more information about the international results: http://www.oecd.org/site/piaac/ . 

•Explore PIAAC data in the International Data Explorer (IDE): http : / / www. oecd.org/ site/ piaac/ 
publicdataandanalysis.htm . 

•Access public-use data files that are available at http : //nces . ed.gov/ surveys/ piaac/ index. asp . 

•Access restricted-use data files that are available to NCES Restricted-use Data Licensees. More 
information on licenses can be found at http://nces.ed.gov/pubsearch/licenses.asp . 
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Appendix A: Members of the PIAAC Subject 

Matter Expert Groups 

This appendix lists the members of the four expert groups responsible for developing the frameworks 
for the PIAAC content domains. 

Literacy Expert Group 

Stan Jones (Chair), Canada 

Egil Gabrielsen, Center for Reading Research, University of Stavanger, Norway 
Jan Hagston, Australia 

Pirjo Linnakyla, University of Jyvaskyla, Finland 

Hakima Megherbi, University of Paris, France 

John Sabatini, Educational Testing Service, United States of America 

Monika Troster, German Institute for Adult Education, Germany 

Eduardo Vidal-Abarca, Department of Psychology, University of Valencia, Spain 

Numeracy Expert Group 

Iddo Gal (Chair), University of Haifa, Israel 

Silvia Alatorre, National Pedagogical University, Mexico 

Sean Close, St. Patricks College, Ireland 

Jeff Evans, Middlesex University, United Kingdom 

Lene Johansen, Aalborg University, Denmark 

Terry Maguire, Institute of Technology Tallaght-Dublin, Ireland 

Myrna Manly, United States of America 

Dave Tout, Australian Council for Educational Research, Australia 

Problem Solving in Technology-Rich Environments 
Expert Group 

Jean-Fran<jois Rouet (Chair), CNRS and University of Poitiers, France 

Mirelle Betrancourt, University of Geneva, Switzerland 

M. Anne Britt, Northern Illinois University, United States of America 

Dr. Rainer Bromme, University of Muenster, Germany 

Arthur C. Graesser, University of Memphis, United States of America 

Jonna M. Kulikowich, Pennsylvania State University, United States of America 

Donald J. Leu, University of Connecticut, United States of America 

Naoki Ueno, Musashi Institute of Technology, Japan 

Herre van Oostendorp, Utrecht University, Netherlands 

Reading Components Expert Group 

John Sabatini, Educational Testing Service, United States of America 
Kelly M. Bruce, Educational Testing Service, United States of America 
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Appendix B: PIAAC Scales and Proficiency- Level 

Descriptions 

This appendix describes in more detail the PIAAC scales and the proficiency-level descriptions that 
accompany these scales. PIAAC proficiency results are also reported in terms of the percentages of 
adults performing at or “reaching” each of the proficiency levels. 

Overview 

PIAAC defines four core competency domains of adult cognitive skills that are seen as key to 
facilitating the social and economic participation of adults in advanced economies: 

• Literacy 

• Reading components 

• Numeracy 

• Problem solving in technology-rich environments 1 

As described in appendix C, PIAAC is administered in either paper-and-pencil mode or via computer 
interface, depending on the assessment domain. Literacy and numeracy are offered in both paper-and- 
pencil and computer modes. Reading components, which are designed to provide information about 
the literacy skills of adults at the lower end of the literacy spectrum, are offered only in paper-and- 
pencil mode. Problem solving in technology-rich environments is administered via computer only. 

The OECD oversees the work of several teams of experts in the development of assessment 
frameworks in each of the domains (see appendix A). Assessment frameworks are available at http:// 
www.oecd.org/site/piaac/publications.htm . Information about the item development and proficiency 
level setting process will be included in a forthcoming PIAAC technical report from OECD. 

Literacy 

The PIAAC literacy framework expands on the definition of literacy previously used in IALS and ALL. 
PIAAC broadly defines literacy as “ understanding, evaluating, using and engaging with written text to 
participate in society ; to achieve ones goals and to develop ones knowledge and potential (Organization for 
Economic Cooperation and Development [OECD] 2012.) 

The purpose of this expanded definition is to highlight the ranges of cognitive processes involved in 
literacy, focus on a more active role of individuals in society, and include various text types, both in 
print and electronic formats, in the measurement of literacy. 


1 These domains were selected based on prior adult assessments of literacy and new developments in the field of adult assessment. 
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PIAAC items include continuous texts (e.g., text in sentences and paragraphs), non-continuous texts 
(e.g., schedules, graphs, and maps), and electronic texts (including hypertext or text in interactive 
environments, such as forms and blogs). Task activities are presented in home, work, and community 
contexts, addressing various purposes adults pursue in their everyday lives. 

Based on the PIAAC framework, literacy tasks include items in paper-and-pencil and computer-based 
delivery modes that cover a range of difficulties — low, middle, and high — to present a comprehensive 
picture of the level of adult literacy skills in each country or region. 

Reading components 

The primary goal of the PIAAC reading components is to provide information about the literacy skills 
of adults at the lower end of the literacy spectrum — specifically, whether they have the foundational 
skills to develop the higher literacy and numeracy abilities necessary to function in society. 

The reading components assessment focuses on elements of reading that are comparable across the 
range of languages in the participating countries and regions: reading vocabulary, sentence comprehen- 
sion, and basic passage comprehension. 

• The reading vocabulary section asks participants to identify the best word that should be 
used to label different graphic illustrations. This task measures whether participants can 
identify common, concrete print words used in everyday adult interactions in the community, 
home, and workplace. It is not meant to determine the vocabulary knowledge (breadth or 
depth) of the participants. 

• The sentence comprehension section asks participants to identify whether sentences of 
varying grammatical/syntactic complexity make sense. This task measures whether partici- 
pants can understand and correctly judge the accuracy of the content of sentences. 

• The basic passage comprehension section asks participants to make a choice between a 
correct and an incorrect word to complete a sentence within a passage. This task measures 
whether respondents comprehend text in context and can appropriately use words in ways 
that characterize fluency. 

The reading component portion of the assessment is optional for countries and regions participating in 
PIAAC. In countries and regions that adopt the reading components tasks, participants who decide not to 
take the computer-based assessment, and those who fail to pass the computer-administered information 
and communication technology (ICT) skills and literacy/numeracy “Core” items, are directed to the 
reading components tasks. (Additional information about the administration of the assessment and the 
“Core” items can be found in appendix C.) Data from the reading components portion of the assessment 
are not reported separately in this First Look report, but can be accessed from the International Data 
Explorer (IDE) at http://www.oecd.org/site/piaac/publicdataandanalysis.htm . 
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Exhibit B-1. Description of PIAAC proficiency levels on the literacy scale: 2012 


Proficiency levels 
and cut scores 
for literacy 

Literacy task descriptions 

Level 5 
(376 - 500) 

At this level, tasks may require the respondent to search for and integrate information across 
multiple, dense texts; construct syntheses of similar and contrasting ideas or points of view; 
or evaluate evidenced based arguments. Application and evaluation of logical and concep- 
tual models of ideas may be required to accomplish tasks. Evaluating reliability of evidentiary 
sources and selecting key information is frequently a key requirement. Tasks often require 
respondents to be aware of subtle, rhetorical cues and to make high-level inferences or use 
specialized background knowledge. 

Level 4 
(326 - 375) 

Tasks at this level often require respondents to perform multiple-step operations to integrate, 
interpret, or synthesize information from complex or lengthy continuous, non-continuous, 
mixed, or multiple type texts. Complex inferences and application of background knowledge 
may be needed to perform successfully. Many tasks require identifying and understanding 
one or more specific, non-central ideas in the text in order to interpret or evaluate subtle 
evidence-claim or persuasive discourse relationships. Conditional information is frequently 
present in tasks at this level and must be taken into consideration by the respondent. Com- 
peting information is present and sometimes seemingly as prominent as correct information. 

Level 3 
(276 - 325) 

Texts at this level are often dense or lengthy, including continuous, non-continuous, mixed, 
or multiple pages. Understanding text and rhetorical structures become more central to suc- 
cessfully completing tasks, especially in navigation of complex digital texts. Tasks require 
the respondent to identify, interpret, or evaluate one or more pieces of information, and often 
require varying levels of inferencing. Many tasks require the respondent construct meaning 
across larger chunks of text or perform multi-step operations in order to identify and formu- 
late responses. Often tasks also demand that the respondent disregard irrelevant or inappro- 
priate text content to answer accurately. Competing information is often present, but it is not 
more prominent than the correct information. 

Level 2 
(226 - 275) 

At this level, the complexity of text increases. The medium of texts may be digital or printed, 
and texts may be comprised of continuous, non-continuous, or mixed types. Tasks in this 
level require respondents to make matches between the text and information, and may 
require paraphrase or low-level inferences. Some competing pieces of information may be 
present. Some tasks require the respondent to 

• cycle through or integrate two or more pieces of information based on criteria, 

• compare and contrast or reason about information requested in the question, or 

• navigate within digital texts to access-and-identify information from various parts 
of a document. 


Level 1 
(176-225) 

Most of the tasks at this level require the respondent to read relatively short digital or print 
continuous, non-continuous, or mixed texts to locate a single piece of information which 
is identical to or synonymous with the information given in the question or directive. Some 
tasks may require the respondent to enter personal information onto a document, in the 
case of some non-continuous texts. Little, if any, competing information is present. Some 
tasks may require simple cycling through more than one piece of information. Knowledge 
and skill in recognizing basic vocabulary, evaluating the meaning of sentences, and reading 
of paragraph text is expected. 

Below Level 1 
(0-175) 

The tasks at this level require the respondent to read brief texts on familiar topics to locate 
a single piece of specific information. Only basic vocabulary knowledge is required, and the 
reader is not required to understand the structure of sentences or paragraphs or make use 
of other text features. There is seldom any competing information in the text and the 
requested information is identical in form to information in the question or directive. While the 
texts can be continuous, the information can be located as if the text were non-continuous. 
As well, tasks below level 1 do not make use of any features specific to digital texts. 
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Exhibit B-2. Examples of literacy items 


Items that exemplify the pertinent features of the proficiency levels in the domain of literacy are 
described below. In order to be consistent with the OECD international report, Levels 4 and 5 are 
combined in the figures in this report (Level 4/5). 

Level 4: Library search (Item ID: C323P002) 

Difficulty score: 348 

The stimulus displays results from a bibliographic search from a simulated library website. The test-taker is 
asked to identify a book suggesting that the claims made both for and against genetically modified foods are 
unreliable. He or she needs to read the title and the description of each book in each of the entries reporting 
the results of the bibliographic search in order to identify the correct book. Many pieces of distracting informa- 
tion are present. The information that the relevant book suggests that the claims for and against genetically 
modified foods are unreliable must be inferred from the statement that the author “describes how both sides in 
this hotly contested debate have manufactured propaganda, tried to dupe the public and... [text ends]”. 

Level 3: Library search (Item ID: C323P003) 

Difficulty score: 289 

The stimulus displays results from a bibliographic search from a simulated library website. The test-taker 
is asked to identify the name of the author of a book called Ecomyth. To complete the task, the test-taker 
has to scroll through a list of bibliographic entries and find the name of the author specified under the 
book title. In addition to scrolling, the test-taker must be able to access the second page where Ecomyth is 
located by either clicking the page number “2” or the word “next”. There is considerable irrelevant infor- 
mation in each entry to this particular task, which adds to the complexity of the task. 

Level 2: Lakeside fun run (Item ID: C322P002) 

Difficulty score: 240 

The stimulus is a simulated website containing information about the annual fun run/walk organized by the 
Lakeside Community Club. The test-taker is first directed to a page with several links, including “Contact 
Us” and “LAQs”. He or she is then asked to identify the link providing the phone number of organizers of 
the event. In order to answer this item correctly, the test- taker needs to click on the link “Contact Us”. This 
requires navigating through a digital text and some understanding of web conventions. While this task might 
be fairly simple for test-takers familiar with web-based texts, some respondents less familiar with web-based 
texts would need to make some inferences to identify the correct link. 
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Level Is Generic medicine (Item ID: C309A321) 

Difficulty score: 219 

The stimulus is a short newspaper article entitled “Generic medicines: Not for the Swiss”. It has two 
paragraphs and a table in the middle displaying the market share of generic medicines in 14 European 
countries and the United States. The test-taker is asked to determine the number of countries in which 
the generic drug market accounts for 10 percent or more of total drug sales. The test-taker has to count 
the number of countries with a market share greater than 10 percent. The percentages are sorted in 
descending order to facilitate the search. The phrase “drug sales”, however, does not appear in the text; 
therefore, the test-taker needs to understand that “market share” is a synonym for “drug sales” in order 
to answer the question. 

Below Level 1: Election results (Item ID: C302BC02) 

Difficulty score: 162 

The stimulus consists of a short report of the results of a union election containing several brief paragraphs 
and a simple table identifying the three candidates in the election and the number of votes they received. 
The test-taker is asked to identify which candidate received the fewest votes. He or she needs to compare the 
number of votes that the three candidates received and identify the name of the candidate who received the 
fewest votes. The word “votes” appears in both the question and in the table and nowhere else in the text. 


B-5 


Numeracy 

The primary goal of PIAAC s numeracy assessment is to evaluate basic mathematical and 
computational skills that are considered fundamental for functioning in everyday work and social 
life. Numeracy in the PIAAC framework is defined as “the ability to access , use , interpret , and com- 
municate mathematical information and ideas , to engage in and manage mathematical demands of 
a range of situations in adult life” (Organization for Economic Cooperation and Development [OECD] 
2012 .) 

The PIAAC numeracy domain is built on previous large-scale assessments of this domain, school- 
oriented assessments, and a review of requirements of workplace skills, adult learning, and mathematics 
and statistics education. The tasks that measure this domain involve managing a situation or solving a 
problem in a practical context — in home, work, or community settings. These tasks ask respondents 
to work with numbers, proportions, measurements, and statistical concepts, and then call for partici- 
pants to compute, interpret, and communicate the results and mathematical content. The situations 
and problems presented in these tasks involve objects or pictures, text, numbers, graphs, and 
technology-based displays. They also require basic mathematical skills in computation, proportions 
and percentages, an understanding of measurement concepts and procedures, and working with simple 
formulas. Respondents also encounter more complex items that require using models to predict future 
needs, and an understanding of basic statistical concepts and displays. 

In addition, PIAAC numeracy assessment items 

• are set in authentic and culturally appropriate contexts, 

• measure different levels of ability, and 

• use the standard measuring systems of the participating country or region. 

Numeracy tasks include items in paper- and-pencil and computer-based delivery modes that cover a range 
of difficulties — low, middle, and high — to present a comprehensive picture of the level of adult numeracy 
skills in each country or region. 
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Exhibit B-3. Description of PIAAC proficiency levels on the numeracy scale: 2012 


Proficiency levels 
and cut scores 
for numeracy 

Numeracy task descriptions 

Level 5 
(376 - 500) 

Tasks in this level require the respondent to understand complex representations and 
abstract and formal mathematical and statistical ideas, possibly embedded in complex 
texts. Respondents may have to integrate multiple types of mathematical information 
where considerable translation or interpretation is required; draw inferences; develop or 
work with mathematical arguments or models; justify, evaluate, and critically reflect upon 
solutions or choices. 

Level 4 
(326 - 375) 

Tasks in this level require the respondent to understand a broad range of mathematical 
information that may be complex, abstract or embedded in unfamiliar contexts. These 
tasks involve undertaking multiple steps and choosing relevant problem-solving strate- 
gies and processes. Tasks tend to require analysis and more complex reasoning about 
quantities and data; statistics and chance; spatial relationships; change, proportions, and 
formulas. Tasks in this level may also require comprehending arguments or communicating 
well-reasoned explanations for answers or choices. 

Level 3 
(276 - 325) 

Tasks in this level require the respondent to understand mathematical information 
which may be less explicit, embedded in contexts that are not always familiar and repre- 
sented in more complex ways. Tasks require several steps and may involve the choice of 
problem-solving strategies and relevant processes. Tasks tend to require the application of 
number sense and spatial sense; recognizing and working with mathematical relationships, 
patterns, and proportions expressed in verbal or numerical form; interpretation and basic 
analysis of data and statistics in texts, tables, and graphs. 

Level 2 
(226 - 275) 

Tasks in this level require the respondent to identify and act upon mathematical information 
and ideas embedded in a range of common contexts where the mathematical content is 
fairly explicit or visual with relatively few distractors. Tasks tend to require the application 
of two or more steps or processes involving calculation with whole numbers and common 
decimals, percents and fractions; simple measurement and spatial representation; estima- 
tion; interpretation of relatively simple data and statistics in texts, tables, and graphs. 

Level 1 
(176-225) 

Tasks in this level require the respondent to carry out basic mathematical processes in 
common, concrete contexts where the mathematical content is explicit with little text and 
minimal distractors. Tasks usually require simple one-step or two-step processes involving 
performing basic arithmetic operations; understanding simple percents such as 50 percent; 
or locating, identifying, and using elements of simple or common graphical or 
spatial representations. 

Below Level 1 
(0-175) 

Respondents classified at this level may have difficulty with many tasks at Level 1 . They 
may be able to cope with very simple tasks set in concrete, familiar contexts where the 
mathematical content is explicit with little or no text or distractors, and that require only 
simple processes such as counting; sorting; performing basic arithmetic operations with 
whole numbers or money, or recognizing common spatial representations. 


Numeracy task descriptions 
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Exhibit B-4. Examples of numeracy items 


Items that exemplify the pertinent features of the proficiency levels in the domain of numeracy are 
described below. In order to be consistent with the OECD international report, Levels 4 and 5 are 
combined in the figures in this report (Level 4/5). No items mapped at level 5 in numeracy. 

Level 4: Education level (Item ID: C632P001) 

Difficulty score: 354 

The stimulus for this item consists of two stacked-column bar graphs presenting the distribution of the 
Mexican population by years of schooling for men and women separately. The y-axis of each of the graphs 
is labeled “percentage” with 6 grid lines labeled “0%”, “20%”, “40%”, “60%”, “80%”, and “100%”. The 
x-axis is labeled “year” and data is presented for I960, 1970, 1990, 2000, and 2005. A legend identifies three 
categories of schooling: “more than 6 years of schooling”, “up to 6 years of schooling”, and “no schooling”. 
The test-taker is asked to approximate what percentage of men in Mexico had more than 6 years of schooling 
in 1970, choosing from a pull-down menu that has 10 response categories: “0-10%”, “10-20%”, and so on. 

Level 3: Package (Item ID: C657P001) 

Difficulty score: 315 

The stimulus for this item consists of an illustration of a box constructed from folded cardboard. The 
dimensions of the cardboard base are identified. The test-taker is asked to identify which plan best 
represents the assembled box out of four plans presented in the stimulus. 

Level 2: Logbook (Item ID: C613A520) 

Difficulty score: 250 

The stimulus for this item consists of a page from a motor vehicle logbook with columns for the date of 
the trip (start and finish), the purpose of the trip, the odometer reading (start and finish), the distance 
travelled, the date of entry and the drivers name and signature. Lor the first date of travel (June 5), the 
column for the distance travelled is completed. The instructions inform the test-taker that “a salesman 
drives his own car and must keep a record of the miles he travels in a Motor Vehicle Log. When he travels, 
his employer pays him $0.35 per mile plus $40.00 per day for various costs such as meals.” The test-taker 
is asked to calculate how much he will be paid for the trip on June 5. 
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Level Is Candles (Item ID: C615A602) 

Difficulty score: 22 1 

The stimulus for this item consists of a photo of a box containing tea light candles. The packaging identi- 
fies the product (tea light candles), the number of candles in the box (105 candles) and its weight. While 
the packaging partially covers the top layer of candles, it can be seen that the candles are packed in five 
rows of seven candles each. The instructions inform the test-taker that there are 105 candles in a box and 
asks him or her to calculate how many layers of candles are packed in the box. 

Below Level 1: Price tag (Item ID: C602A501) 

Difficulty score: 168 

The stimulus for this item consists of four supermarket price tags. These identify the product, the price per 
pound, the net weight, the date packed and the total price. The test-taker is asked to indicate the item that 
was packed first by simply comparing the dates on the price tags. 
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Problem solving in technology-rich environments 

PIAAC represents the first attempt to assess problem solving in technology-rich environments on a 
large scale and as a single dimension in an international context. PIAAC defines problem solving in 
technology-rich environments as “using digital technology, communication tools, and networks to 
acquire and evaluate information, communicate with others, and perform practical tasks” 

(Organization for Economic Cooperation and Development [OECD], 2012.) 

Digital technology has revolutionized access to information and communication capabilities over the 
past two decades. In particular, the Internet has increased instantaneous access to large amounts of 
information and has expanded instant voice, text, and graphics capabilities across the globe. In order to 
effectively operate in these environments, it is necessary to have 

• knowledge of how various technological environments are structured (e.g., an understanding 
of the basics of the environment, including how to use command names, drop-down menus, 
naming protocols for files and folders, and links in a web page); and 

• the ability to interact effectively with digital information; understand electronic texts, images, 
graphics, and numerical data; and locate, evaluate, and critically judge the validity, accuracy, 
and appropriateness of the accessed information. 

These skills constitute the core aspects of the problem solving in technology-rich environments domain. 

Items in this domain present tasks of varying difficulty in simulated software applications using 
commands and functions commonly found in email, web pages, and spreadsheets. These tasks range 
from purchasing particular goods or services online and finding interactive health information to 
managing personal information and business finances. 

PIAAC recognizes the diversity of digital technologies and the fact that they are evolving at a rapid 
pace, but due to implementation constraints, the first round of PIAAC was limited to using com- 
puters and simulated computer networks. The tasks assessing problem solving in technology-rich 
environments were only administered via computer and therefore only those taking the computerized 
assessment received a score in this domain. 
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Exhibit B-5. Description of PIAAC proficiency levels on the problem solving in 
technology-rich environments scale: 2012 


Proficiency levels 
and cut scores 
for problem 
solving in 
technology-rich 
environments 

Problem solving in technology-rich environments task descriptions 

Level 3 
(341 - 500) 

At this level, tasks typically require the use of both generic and more specific technology 
applications. Some navigation across pages and applications is required to solve the 
problem. The use of tools (e.g., a sort function) is required to make progress toward the 
solution. The task may involve multiple steps and operators. In terms of cognitive process- 
ing, the problem goal may have to be defined by the person, and the criteria to be met 
may or may not be explicit. There are typically high monitoring demands. Unexpected 
outcomes and impasses are likely to occur. The task may require evaluating the relevance 
and the reliability of information in order to discard distractors. Integration and inferential 
reasoning may be needed to a large extent. 

Level 2 
(291 - 340) 

At this level, tasks typically require the use of both generic and more specific technology 
applications. For instance, the person may have to make use of a novel online form. Some 
navigation across pages and applications is required to solve the problem. The use of tools 
(e.g., a sort function) can facilitate the resolution of the problem. The task may involve 
multiple steps and operators. In terms of cognitive processing, the problem goal may have 
to be defined by the person, though the criteria to be met are explicit. There are higher 
monitoring demands. Some unexpected outcomes or impasses may appear. The task may 
require evaluating the relevance of a set of items to discard distractors. Some integration 
and inferential reasoning may be needed. 

Level 1 
(241 - 290) 

At this level, tasks typically require the use of widely available and familiar technology 
applications, such as email software or web browser. There is little or no navigation 
required to access the information or commands required to solve the problem. The prob- 
lem may be solved regardless of one’s awareness and use of specific tools and functions 
(e.g., a sort function). The task involves few steps and a minimal number of operators. At 
a cognitive level, the person can readily infer the goal from the task statement; problem 
resolution requires one to apply explicit criteria; there are few monitoring demands (e.g., 
the person does not have to check whether they have used the adequate procedure or 
made progress toward the solution). Identifying contents and operators can be done 
through simple match; only simple forms of reasoning (e.g., assigning items to categories) 
are required; there is no need to contrast or integrate information. 

Below Level 1 
(0 - 240) 

Tasks are based on well-defined problems involving the use of only one function within a 
generic interface to meet one explicit criterion without any categorical, inferential reasoning 
or transforming of information. Few steps are required and no subgoal has to be generated. 


Problem solving in technology-rich environments task descriptions 
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Exhibit B-6. Examples of problem solving in technology-rich environments items 


Items that exemplify the pertinent features of the proficiency levels in the domain of problem solving in 
technology-rich environments are described below. 

Level 3: Meeting Rooms (Item ID: U02) 

Difficulty score: 346 

This task involves managing requests to reserve a meeting room on a particular date using a reservation system. 
Upon discovering that one of the reservation requests cannot be accommodated, the test-taker has to send an 
email message declining the request. Successfully completing the task involves taking into account multiple 
constraints (e.g., the number of rooms available and existing reservations). Impasses exist, as the initial con- 
straints generate a conflict (one of the demands for a room reservation cannot be satisfied). The impasse has 
to be resolved by initiating a new sub-goal, i.e., issuing a standard message to decline one of the requests. Two 
applications are present in the environment: an email interface with a number of emails stored in an inbox 
containing the room reservation requests, and a web-based reservation tool that allows the user to assign rooms 
to meetings at certain times. The item requires the test-taker to use information from a novel web application 
and several email messages, establish and apply criteria to solve a scheduling problem where an impasse must 
be resolved, and communicate the outcome. The task involves multiple applications, a large number of steps, a 
built-in impasse, and the discovery and use of ad hoc commands in a novel environment. The test-taker has to 
establish a plan and monitor its implementation in order to minimize the number of conflicts. In addition, the 
test-taker has to transfer information from one application (email) to another (the room-reservation tool). 

Level 2: Club Membership (Item ID: U19b) 

Difficulty score: 296 

This task involves responding to a request for information by locating information in a spreadsheet and 
emailing the requested information to the person who asked for it. The test-taker is presented with a 
word-processor page containing a request to identify members of a bike club who meet two conditions, 
and a spreadsheet containing 200 entries in which the relevant information can be found. The required 
information has to be extracted by using a sort function. The item requires the test-taker to organize large 
amounts of information in a multiple-column spreadsheet using multiple explicit criteria and locate and 
mark relevant entries. The task requires switching between two different applications and involves multiple 
steps and operators. It also requires some amount of monitoring. Making use of the available tools greatly 
facilitates identifying the relevant entries. 

Level 1: Party Invitations (Item ID: U01A) 

Difficulty score: 286 

This task involves sorting emails into pre-existing folders. An email interface is presented with five emails 
in an inbox. These emails are responses to a party invitation. The test-taker is asked to place the response 
emails into a pre-existing folder to keep track of who can and cannot attend a party. The item requires the 
test-taker to categorize a small number of messages in an email application in existing folders according 
to a single criterion. The task is performed in a single and familiar environment and the goal is explicitly 
stated in operational terms. Solving the problem requires a relatively small number of steps and the use 
of a restricted range of operators and does not demand a significant amount of monitoring across a large 
number of actions. 
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Appendix C: Methodology and Technical Notes 

This appendix describes the assessment design, sampling, data collection, weighting, and variance 
estimation, scaling, and statistical testing procedures used to collect and analyze the data for PIAAC. 
PIAAC s household data collection was conducted from August 25, 2011 through April 3, 2012. 

Assessment Design 

The PIAAC psychometric assessment design was complex because the assessment measured four 
domains — literacy, numeracy, reading components, and problem solving in technology-rich environ- 
ments — across two modes of administration — paper-and-pencil and computer instruments. In sum- 
mary, the PIAAC psychometric assessment design provided for the following information: 

• Reading component skills among lower performing adults in each participating country, as well 
as among those who reported not knowing how to use a computer. 

• Population distributions in literacy, which could be linked to the International Adult Literacy 
Survey (IALS) and Adult Literacy and Lifeskills Survey (ALL). 

• Population distributions in numeracy, which could be linked to ALL. 

•Accurate estimates of population distributions and a baseline measure of problem solving in 
technology-rich environments for future estimation of trends over time. 

• Insights into strategies and processes that adults use when they responded to the tasks on 
problem solving in technology-rich environments. 

• Pairwise covariance estimates among the various measures, including the relationships between 
literacy and numeracy, literacy and reading components skills, literacy and problem solving in 
technology-rich environments, numeracy and reading components skills, and numeracy and 
problem solving in technology-rich environments. 

• Information that could be used to analyze the relationship between the measured competencies 
and the PIAAC behavioral measures and social/economic measures (from the responses to the 
background questions and job requirements approach module, explained in “Overview of the 
Skill Use Scales in PIAAC”). 

PIAAC was designed as a computer-based assessment. Respondents who had little or no familiarity 
with computers, however, were directed to a pencil-and-paper version of the assessment that tested 
skills in the domains of literacy and numeracy only. Approximately 1 5 percent of the respondents were 
directed to the paper-and-pencil path. Regardless of whether they took the assessment in the computer 
or pencil-and-paper format, all respondents first took a “Core” test to assess their capacity to undertake 
the full assessment. Those who were unsuccessful at the Core test were directed to the assessment of 
reading components. Those who succeeded at the Core test proceeded to the full assessment. 

The PIAAC assessment for the United States included an adaptive element that allowed for automatic 
scoring. Based on their performance at different points in the assessment, respondents taking the 
computer-based version were directed to different “testlets” that contained items of different average 
difficulty in the domains of literacy and numeracy. 
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Sampling 

The 2012 PIAAC assessment included a nationally representative probability sample of 9,468 
households. This household sample was selected on the basis of a four-stage, stratified area sample: 

(1) primary sampling units (PSUs) consisting of counties or groups of contiguous counties; 

(2) secondary sampling units (referred to as segments) consisting of area blocks; (3) housing units 
containing households; and (4) eligible persons within households. Person-level data were collected 
through a screener, a background questionnaire, and the assessment. The screener instrument was 
conducted using a computer- assisted personal interviewing system (CAPI) and collected information 
that included age and gender of all household members. It determined which household member or 
members was/were eligible for the study and selects the sample person(s). Of the 9,468 sampled house- 
holds, 1,283 were either vacant or not a dwelling unit, resulting in a sample of 8,183 households. In 
the sample, there were 1,267 households without an adult age 16 to 65. A total of 5,686 of the 6,916 
households with eligible adults completed the screener (up to two adults per household could be 
selected to complete the questionnaire), which was used to select survey respondents. The final 
screener response rate was 86.5 percent weighted. 

Based on the screener data, 6,100 respondents age 16 to 65 were selected to complete the background 
questionnaire and the assessment; 4,898 actually completed the background questionnaire. Of the 
1,202 respondents who did not complete the background questionnaire, 112 were unable to do so 
because of a literacy-related barrier: either the inability to communicate in English or Spanish (the 
two languages in which the background questionnaire was administered) or a mental disability. Twenty 
others were unable to complete the questionnaire due to technical problems. The final response rate 
for the background questionnaire — which included respondents who completed it and respondents 
who were unable to complete it because of a language problem or mental disability — was 
82.2 percent weighted. 

Of the 4,898 adults age 16 to 65 who completed the background questionnaire, 4,820 completed the 
adult literacy assessment. An additional 22 were unable to complete the assessment for literacy-related 
reasons. Another 1 1 were unable to do so due to technical problems. The final response rate for the 
overall assessment — which included respondents who answered at least one question on each scale and 
the 22 respondents who were unable to do so because of a language problem, mental disability, or tech- 
nical problem — was 99.0 percent weighted. 

As described by PIAAC Technical Standard 4.3.3, a completed case contained at least the following: 

• Responses to key background questions, including age, gender, highest level of schooling, and 
employment status; and 

•A completed Core instrument 1 (i.e., the interviewer asked the respondent all Core questions or 
the Core instrument was not completed for a literacy- related reason [e.g., because of a language 
difficulty] or because the respondent was unable to read or write in any of a country’s PIAAC 
official languages); or 

• Responses to age and gender for literacy-related nonrespondents to the background 
questionnaire/job Requirements Approach (explained in “Overview of the Skill Use Module 
Scales in PIAAC”). 

Cases that experienced technical problems during the administration of the background questionnaire 
or the assessment were also counted as completed. To avoid penalizing countries for platform configura- 


1 For a full description of the Core instrument, see the section on Data Collection. 
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tion issues with the background questionnaire and assessment software package, interviews that could 
not be completed due to software issues were considered “completes” for the purposes of response rate 
calculations. The overall weighted response rate for the household sample was 70.3 percent. 

Table C-1 . Weighted response rate for the United States, by survey component 


Component Percentage 

Screener (household) 86.5 

Background questionnaire 82.2 

Assessment (without reading component) 99.0 

Overall rate (product of component response rates) 70.3 


For respondents who did not complete any tasks on any of the literacy scales, no information is 
available about their performance on the literacy items they were missing. Completely omitting these 
individuals from the analyses would have resulted in unknown biases in estimates of the literacy skills 
of the national population because refusals cannot be assumed to have occurred randomly. For 
respondents who answered the background questionnaire but refused to complete the assessment 
for reasons other than language issues or a mental disability, proficiency values were imputed based on 
the covariance information from those who completed the survey. 

The final household reporting sample — including the imputed cases — consisted of 3,010 respondents. 
These 5,010 respondents are the 4,898 respondents who completed the background questionnaire, 
plus the 112 respondents who were unable to complete the background questionnaire for literacy- 
related reasons. 

The sample was subject to unit nonresponse from the screener, background questionnaire, assessment 
(including reading components), and item nonresponse to background questionnaire items. Although 
the screener had a unit response rate above 85 percent, the background questionnaire had a unit response 
rate below 85 percent and thus required an analysis of the potential for nonresponse bias according to the 
National Center for Education Statistics (NCES) statistical standards. 

Nonresponse Bias 

The nonresponse bias analysis of the household sample revealed differences in the characteristics of 
respondents who participated in the background questionnaire compared with those who refused. In 
a bivariate unit-level analysis at the background questionnaire stage, estimated percentages for respon- 
dents were compared with those for the total eligible sample to identify any potential bias owing to 
nonresponse. Multivariate analyses were conducted to further explore the potential for nonresponse bias 
by identifying the domains with the most differential response rates. These analyses revealed that the 
subgroup with the lowest response rates for the background questionnaire had the following character- 
istics: (1) Hispanic, (2) age 26 and older with no children in the household, and (3) reside outside the 
Northeastern United States in areas with low levels of linguistic isolation (a low percentage who have 
some difficulty speaking English) and with unemployment rates exceeding approximately 5 percent. 

In general, younger persons were found to be more available to participate, as were those with children 
ages 16 and younger, and women. However, the variables found to be significant in the bivariate 
analysis — those used to define areas with low response rates — were used in weighting adjustments. 

The analysis showed that weighting adjustments were highly effective in reducing the bias. The general 
conclusion was that the potential amount of nonresponse bias attributable to unit nonresponse at the 
background questionnaire stage was likely to be negligible. 
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Data Collection 


Whenever possible, interviewers administered the background questionnaire and assessment in a private 
setting (e.g., home or library). Using the computerized interview and assessment software provided by 
the PIAAC Consortium 2 , the interviewer read the background questionnaire questions from a laptop 
and entered all responses directly into the laptop. Skip patterns and follow-up probes for contradictory 
or out-of-range responses were programmed into the interview software. At the completion of the back- 
ground questionnaire, the participant was administered the computer-based Core or the paper-and- 
pencil based Core if the participant could not or would not use the computer. Upon the completion 
and scoring of the Core tasks, the respondent was routed to the computer-based assessment (CBA), the 
paper-based assessment (PBA) of literacy and numeracy, or the paper-based reading components. The 
background questionnaire and the assessment took approximately two hours to complete, however the 
time varied by the respondent. The number of assessment items also varied based on the respondents’ 
performance on the Core and the adaptive routing implemented in the automated portion of the 
assessment. 

The progress through the assessment was controlled by the computer based on the respondent’s 
performance on various components of the assessment. The PIAAC assessment was composed 
of the following components: 

•The Core consisted of three modules: the CBA Core Stage 1, the CBA Core Stage 2, and the 
PBA Core. 

o The CBA Core Stage 1 included six tasks and was designed to determine if the participant 
had the basic set of skills needed to complete the computer-based assessment. To pass the 
CBA Core Stage 1 , the participant needed to correctly answer at least three of the first five 
tasks, plus the sixth task (highlighting text). CBA Core Stage 1 questions were automatically 
scored by the computer, and a participant who passed the CBA Core Stage 1 continued on 
to the CBA Core Stage 2. A participant who did not pass the CBA Core Stage 1 was routed 
to the PBA Core. 

o The CBA Core Stage 2 included six tasks that measured basic literacy and numeracy skills 
necessary to undertake the assessment. CBA Core Stage 2 questions were automatically 
scored by the computer, and a participant who passed the CBA Core Stage 2 continued on 
to the computer-based assessment. A participant who did not pass the CBA Core Stage 2 
was routed directly to the paper-based reading components section. 

o The PBA Core consisted of eight tasks and measured basic literacy and numeracy skills 
necessary to undertake the assessment. PBA Core questions were interviewer scored and 
entered into the computer to determine if the participant passed the PBA Core. A partici- 
pant who passed the PBA Core continued on to the paper-based assessment of literacy and 
numeracy and then to the paper-based reading components section. A participant who did 
not pass the PBA Core was routed directly to the reading components section. 


2 The PIAAC Consortium includes the following organizations: Educational Testing Service (ETS), Westat, cApStAn, the Research Centre for Education and the 
Labor Market (ROA), gesis-ZUMA Centre for Survey Research, German Institute for International Education Research (DIPF), and the Data Processing Centre 
of the International Association for the Evaluation of Educational Achievement (IEA). In addition to these organizations, PIAAC is aided by numerous national 
contracting partners. 
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•The assessment was administered in CBA and PBA modes. 

o The CBA consisted of three “testlets” of tasks at Stage 1 (9 items) and four “testlets” at 
Stage 2(11 items). Each respondent completed two testlets that included items from two 
of the three domains. 

o The PBA consisted of two paper-based assessment booklets, one contained literacy items 
and one contained numeracy items. Each booklet contained 20 items for the participant 
to complete and each participant completed only one booklet type. 

• The reading components were completed by a participant after completing the literacy or 
numeracy booklet. Reading components were also completed by a respondent who failed the 
CBA Core Stage 2 or the PBA Core. 

Problem Solving in Technology-Rich Environments: 

U.S. Sample 

The PIAAC assessment design was developed to route respondents to the most appropriate delivery 
mode as a means to help assure the most reliable, valid, and comparable assessment of skills. The 
computer-based assessment (CBA) was chosen for those demonstrating information and communication 
technology (ICT) skills, while the remaining respondents received the paper-based assessment (PBA). 
The scores for respondents that had no computer experience, failed the ICT skills test, or refused the 
CBA did not contribute to the estimation of the item parameters for the problem solving in technology- 
rich environments domain. The design of the PIAAC assessment contained only literacy and numeracy in 
the PBA because the problem solving in technology-rich environments assessment, by definition, was 
suitable only for respondents familiar with ICT environments. Exhibit C-l illustrates the stages of the 
assessment administration and the weighted percentages of U.S. respondents at each stage of the assessment. 

Exhibit C-l. PIAAC Main Study Yield 
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Weighting and Variance Estimation 

A complex sample design was used to select assessment respondents. The properties of a sample selected 
through a complex design could be very different from those of a simple random sample in which every 
individual in the target population has an equal chance of selection and in which the observations from 
different sampled individuals can be considered statistically independent of one another. Therefore, 
the properties of the sample for the complex data collection design were taken into account during the 
analysis of the data. 

One way of addressing the properties of the sample design was by using sampling weights to account for 
the fact that the probabilities of selection were not identical for all respondents. The sampling weights 
were further adjusted for nonresponse to the screener and background questionnaire, extreme weights 
were trimmed, and weights for all respondents calibrated to the U.S. Census Bureaus 2010 American 
Community Survey population totals for those age 16 to 65. Since literacy-related nonrespondents to 
the screener, the background questionnaire, and the assessment are similar in proficiency, the weights 
of the literacy- related nonresponse cases were not adjusted during the screener-level nonresponse adjust- 
ment. Instead, the background questionnaire weights for the background questionnaire and assessment 
literacy-related cases were adjusted to account for the literacy-related screener nonrespondents. This 
adjustment was necessary primarily to allow the literacy-related background questionnaire and assessment 
nonrespondents to represent the literacy-related screener nonrespondents in the calibration procedure. 

All population and subpopulation characteristics based on the PIAAC data used sampling weights in their 
estimation. The statistics presented in this report are estimates of group and subgroup performance based 
on a sample of respondents, rather than the values that could be calculated if every person in the nation 
answered every question on the instrument. Therefore, it is important to have measures of the degree of 
uncertainty of the estimates. Accordingly, in addition to providing estimates of percentages of respondents 
and their average scale scores, this report provides information about the uncertainty of each statistic in 
the form of standard errors on the U.S. PIAAC website at http:// nces.ed.gov/ surveys/ piaac/ results/ 
summary, aspx . 

Because the assessment used clustered sampling, conventional formulas for estimating sampling 
variability (e.g., standard errors) that assume simple random sampling and hence independence of 
observations would have been inappropriate for this report. For this reason, the PIAAC assessment 
used a paired jackknife replication approach (sometimes referred to as JK2) to estimate standard errors 
(Rust and Rao, 1996). 

Overview of the Skill Use Scales in PIAAC 

In PIAAC, the skills of a population were measured not only directly through the cognitive 
instruments, but also indirectly through the background questionnaire that asked respondents 
to report on their use of skills both in and outside of work. The frequency and type of activities 
associated with reading, writing, numeracy, and information technology were targeted in the back- 
ground questionnaire using multiple items that were similarly worded to apply to activities both in and 
outside of work. In addition, other areas, particularly those involving intrapersonal, interpersonal, and 
other generic “soft” skills not included in the direct assessment, were addressed through a set of self- 
reported questions. Examples of some of these questions include how individuals deal with new informa- 
tion and the extent to which individuals like to learn new things. This set of questions made up a module 
within the questionnaire that was specifically developed for the PIAAC project: the Job Requirements 
Approach (JRA) Module (Green 2009). When the PIAAC Consortium developed and implemented 
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PIAAC, they were able to construct 13 scales based on a cross-country analysis of comparability 
reliability and convergent as well as discriminant validity The 13 scales are: 


• ICT at home 

• ICT at work 

• Influence 

• Learning at work 

• Numeracy at home 

• Numeracy at work 

• Planning 


• Reading at home 

• Reading at work 

• Readiness to learn 

• Task discretion 

• Writing at home 

• Writing at work 


These scales were constructed using item response theory (IRT), more specifically the generalized partial 
credit model (GPCM), and person-specific levels of skill use were estimated using weighted likelihood 
estimation (WLE; Warm 1989, Muraki 1992, Penfield & Bergeron 2005). Scale values were derived 
for all respondents who reported at least some limited activities in each of these domains. Those who 
reported no skill use in each of the 13 areas are not represented on any of these 13 scales. Nevertheless, 
they provide important information with respect to the percentage of people in each participating 
country who do not use particular types of skills either in or outside of work. 


Skill Use Derived Variables 

The skill use variables reported on page 1 8 consist of respondents’ answers to multiple questions in the 
background questionnaire. Reading skills used at work include respondents’ answers to questions 
regarding frequency of engagement in the following activities: read directions or instructions; read let- 
ters, memos or emails; read articles in newspapers, magazines or newsletters; read articles in professional 
journals or scholarly publications; read manuals or reference materials; read bills, invoices, bank state- 
ments or other financial statements; read diagrams, maps or schematics. Numeracy skills used at work 
include respondents’ answers to questions regarding frequency of engagement in the following activi- 
ties: calculate prices, costs or budgets; use or calculate fractions, decimals or percentages; use a calcula- 
tor; prepare charts, graphs or tables; use simple algebra or formulas; use more advanced math or statis- 
tics such as calculus, complex algebra, trigonometry or use of regression techniques. ICT skills 3 used at 
work includes respondents’ answers about the frequency with which they use email; use the Internet to 
understand issues related to work; use of a word processor; use of programming language to program or 
write computer code; participate in real-time discussions on the Internet. The respondents’ answers to 
questions regarding the frequency use of these skills range from “never,” “less than once a month,” “less 
than once a week but at least once a month,” “at least once a week but not every day,” to “Every day.” 
Respondents who are in the higher quintiles used these skills more often than the respondents who are 
in the lower quintiles. 


3 ICT skills refers to information and communication technology s ki lls. 
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Scaling 

Information on scaling in the PIAAC assessment can be found on the OECD PIAAC website at 
http://www.oecd.org/site/piaac/ . 

Statistical Testing 

The statistical comparisons in this report were based on the t statistic. Statistical significance was 
determined by calculating a t value for the difference between a pair of means or proportions, and 
comparing this value with published tables of values at a certain level of significance, called the alpha 
level. The alpha level is an a priori statement of the probability of inferring that a difference exists when, 
in fact, it does not. Findings from £-tests are reported based on a statistical significance (or alpha level) 
set at .05, without adjustments for multiple comparisons. 
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